{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**内容:**\n",
    "- 逻辑回归算法原理\n",
    "- 用梯度下降算法求解逻辑回归算法的模型参数\n",
    "- 正则化及正则化应用\n",
    "- L1范数和L2范数的含义及其作为模型正则项的区别\n",
    "- 用逻辑回归算法求解乳腺癌检测问题"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.算法原理\n",
    "\n",
    "1.预测函数:  \n",
    "需要找出一个预测函数模型,使其值输出在[0,1]之间,然后选择一个基准值,如0.5,如果算出来的预测值大于0.5,就认为预测值是1,反之认为其是0,选择:\n",
    "$$\n",
    "g(x)=\\frac{1}{1+e^{-z}}\n",
    "$$\n",
    "来作为其预测函数,其中e是自然对数的底数.函数g(z)称为Sigmoid函数,也称为Logistic函数.以z为横坐标,以g(z)为纵坐标,画出图形如下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hide_input": true
   },
   "outputs": [],
   "source": [
    "sns.set_style('whitegrid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hide_input": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD7CAYAAACVMATUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XlclHXiB/DPHMxwDIeICqgoogimiGhZGZoRnh2blpBHuenuZu36Sk07tszlZx5lZfXyyjZq2zaPDrMyU9SycNeDACUHEFAURUHuGZjzeX5/4E5rqaMDwzPH5/16+WKemXmYz1fgw5dnnkMmiqIIIiLyCHKpAxARUfthqRMReRCWOhGRB2GpExF5EJY6EZEHYakTEXkQljoRkQdhqRMReRCWOhGRB1F29Avm5eVBrVY7tK7RaHR4XVfDsbgeTxkHwLG4qraMxWg0IjEx0e7zOrzU1Wo14uPjHVpXq9U6vK6r4Vhcj6eMA+BYXFVbxqLVaq/redz8QkTkQVjqREQehKVORORBWOpERB6EpU5E5EGuq9Tz8/MxY8aM39y/d+9eTJ48GWlpadiyZUu7hyMiohtjd5fGjRs3Yvv27fDz87vsfrPZjOXLl+OTTz6Bn58fHn74YYwePRpdunRxWlgiIro2u6UeFRWFt99+G4sWLbrs/tLSUkRFRSE4OBgAMHToUBw5cgTjx493TlIiIolYBREtZitaTFYYzFbbbZNVgNkiwGQVYLIIMFtFmKxWmC3i/9zX+k8ul2FYiNXpWe2W+tixY1FRUfGb+3U6HQIDA23LAQEB0Ol0dl/QaDRe9070v2YwGBxe19VwLK7HU8YBcCy2dS0CGgxWNBkF6E0C9OZLH39922RFs1mE0SLAYBVhtFy6bWm9bRbafilnlUKG/xvdGYFO/ro4fESpRqOBXq+3Lev1+stK/mp4RGkrjsX1eMo4AM8eiyCIuKg3orLegMqGFlQ2GHC+0YAanQm1ehNqdEbU6FtvN5uuPTMOUCkQ5OeDQF8lAtRKdApQwM9HCT+VAn4+cvj5KOCnUl762Lrs66OAv0oJXx85VEo5fBStH1WKX5Z9FDLbff99XCmXobCw0OlHlDpc6jExMSgvL0d9fT38/f1x5MgRzJo1y9FPR0Rk09BixqmLepyq0eOwtg66/FycqzfgXEMLLjQaYLZePnNWKeTorFEhNECFzho1+nTRXLqtQucAFUL8VQjybS3v4EslrlEroVR43g6AN1zqX375JZqbm5GWloZnn30Ws2bNgiiKmDx5Mrp16+aMjETkofRGCwrPN0Fb2YjC840orGzCyYt61OhNtufIAESGGNC9kx+G9eqE8GA/RIb4IiLYDxHBvogI9kVogAoymUy6gbiQ6yr1Hj162HZZvPfee23333XXXbjrrruck4yIPIrBbEXB2QbklNchv6Iex881ory2GeKlSXegWom4iECkDuiG6LAA9A4LQHRYAFqqT2PwwJukDe9GOvwsjUTkHfRGCw6dqkX2iYs4XF6H4+cabJtNokL9cVNkECYl9UBceCDiI4LQo5PfFWfb2lrP20TiTCx1ImoXoiji53ON2FtYhR9LLiL3dB3MVhEqpRyJPUMw644+SIoKQVKvTgjTeMb50V0RS52IHGa2Cjh8sha7jl/Arp/P41yDATIZMDAyGLPu6IM7+oZhWO9O8PVRSB3Va7DUieiGiKKI/IoGfPZTBb7MP4e6ZjPUSjlGxnbBvNRY3BXXFZ05E5cMS52IrsuFRgM+yanAZz9VoLRaD7VSjjE3heOehAgk9wuDv4p14gr4VSCia8o9XYfM7FPYcawSFkHELb1D8ceRfTB+UASCfH2kjke/wlInot+wWAXsKDiP9348ibwz9QhUK/Ho7b0x49Ze6B0WIHU8ugaWOhHZWAURXx09hzezTqDsoh7RYQH42303YfLQHtCoWRfugF8lIoIgiNhRUInVWSdQUqVDXHgg1k9PwpgB4ZDLeaSmO2GpE3m5/5TVIOPL4zhe2Yh+XTVYOy0J425imbsrljqRlzpT24zl32ix49h5dA/xw5vpibgnIRIKlrlbY6kTeRmLVcDffzyJN7KKIYMMC1Jj8YeRfXiAkIdgqRN5kYKzDXjm06P4+VwjUgd0Q8b9NyEi2M/+iuQ2WOpEXsBiFbDuu1K8uecEOgWosG5aEsYNDOfpaj0QS53Iw52uaca8LXnIKa/DfYMj8X/3D0SwPw8a8lQsdSIPtrOgEk9vPQqZDHgzPRH3J3aXOhI5GUudyANZBBEvf30cG384icE9Q7Bm6hD06OQvdSzqACx1Ig9TozPiuV2VKLhgwCO39cJfJ8ZDreSeLd6CpU7kQUqqmvDY+0dwvsGIN9IG44EhPaSORB2MpU7kIX48cRFzPsqBWinHyrERLHQvxYv/EXmAbblnMTPzECKD/bDtyRGI6+IrdSSSCEudyM19cOAUntqch2G9O+GTObfxDVEvx80vRG5KFEW8tacEb2QVI3VAN7z98BAe6k8sdSJ3JIoiVu0qwpp9pZic1AMrJw+CUsE/vImlTuSW3thdjDX7SvHwLT3x8u8G8TS5ZMNf7URuZnVWMd7aW4K0YSx0+i2WOpEb2bi/DKuzTuChoT2wfBILnX6LpU7kJj7NqcDLO7SYmBCBFZMTWOh0RSx1Ijewt/ACFn16FHf0DcPrUwbz6kR0VSx1IheXe7oOT3z0E26KDML6GUN5Hhe6JpY6kQurqGvGH/6Rg66Bvnhv5s3QqLnDGl2b3VIXBAGLFy9GWloaZsyYgfLy8sse//vf/45JkyZh8uTJ2L17t9OCEnkbndGC2R8cgdFixXszhyFMo5Y6ErkBu7/2s7KyYDKZsHnzZuTl5WHFihVYt24dAKCxsREffvghdu3ahZaWFvzud79Damqq00MTeTqrIGLux7k4UaVD5syb0bdroNSRyE3Ynann5OQgOTkZAJCYmIiCggLbY35+foiMjERLSwtaWlp4vUOidrJqVxH2FlZhyb0DMDK2i9RxyI3YnanrdDpoNBrbskKhgMVigVLZumpERAQmTpwIq9WKP/3pT3Zf0Gg0QqvVOhTWYDA4vK6r4Vhcj6uMI7tcj3XfXcD42EAMC2lxKJOrjKU9cCw3xm6pazQa6PV627IgCLZC379/P6qqqrBnzx4AwKxZs5CUlISEhISrfj61Wo34+HiHwmq1WofXdTUci+txhXGUVOmwelM2BvcMwepHbnV4TxdXGEt74Vh+Wfd62N38kpSUhP379wMA8vLyEBsba3ssODgYvr6+UKlUUKvVCAwMRGNjo0OBibyd3mjB4//MgUopx7ppSdx1kRxid6aempqK7OxspKenQxRFLFu2DJmZmYiKikJKSgoOHDiAKVOmQC6XIykpCSNGjOiI3EQe54VtBSir1uGfs4YjMsRP6jjkpuyWulwuR0ZGxmX3xcTE2G7PnTsXc+fObf9kRF7k05wKfJ57FvPujsXtfcOkjkNujAcfEUmsrFqHF78owPDoUPz5rr5SxyE3x1InkpDRYsVfPs6FSinH6vREntOF2ozHHBNJ6LVdxfj5XCM2PjIMEcHcjk5tx5k6kUSOnKrFxh/KMHV4FFIHdJM6DnkIljqRBJpNFizYmo/uIX54foJn7INNroGbX4gksPKbQpTXNGPTH2/lmRepXXGmTtTBDpRcxAf/LsfvR/TGrX06Sx2HPAxLnagDNRnMWPjJUUSHBWDR2Dip45AH4t99RB3o5a+1qGxowdbHb4efiqcBoPbHmTpRBzlQehGbDp/BH0b2wdBenaSOQx6KpU7UAYwWK174vAC9Ovtj3t2x9lcgchA3vxB1gHXflaLsoh7/eOwW+Ppwsws5D2fqRE5WWq3D2n2luG9wJK9iRE7HUidyIlEU8eK2Aqh95HjhHh5kRM7HUidyos9zz+JAaQ2eHR+HroG+UschL8BSJ3KSOr0JS7/WIikqBA/fHCV1HPISLHUiJ3nl20I0tJjx8gODIOcpdamDsNSJnKDgbAM2HT6Dmbf3RnxEkNRxyIuw1InamSiKWLL9Z4T6qzA3pZ/UccjLsNSJ2tn2/HM4Ul6HhWP7I9jPR+o45GVY6kTtqNlkwYpvCjGwexAeGtZT6jjkhXhEKVE7Wv9dKSobDHj74SG83ihJgjN1onZyprYZG/aX4f7ESAzrHSp1HPJSLHWidrL8Gy3kMhmeHc/zpJN0WOpE7eBgWQ12HDuPJ+6MQUSwn9RxyIux1InaSBBELNuhRUSwL/4wso/UccjLsdSJ2uirY5XIr2jAgjH9eVpdkhxLnagNjBYrXv22EPERQXhgSHep4xCx1Ina4sN/l+NMbQuenxDHXRjJJbDUiRzU0GzG23tLkNwvDMn9ePELcg0sdSIHrfmuBI0GM54bz4tfkOtgqRM54ExtM97PPoVJQ3pgQCTPwkiuw+5pAgRBwJIlS1BUVASVSoWlS5eiV69etse///57rFmzBgAwYMAAvPTSS5DJuG2RPNvru4shkwFPj42VOgrRZezO1LOysmAymbB582YsWLAAK1assD2m0+nw6quvYv369diyZQu6d++Ouro6pwYmklrR+SZsyzuL34+I5oFG5HLslnpOTg6Sk5MBAImJiSgoKLA9lpubi9jYWKxcuRJTp05FWFgYQkN5zgvybK/vLoJGpcTjo3igEbkeu5tfdDodNBqNbVmhUMBisUCpVKKurg4HDx7Etm3b4O/vj2nTpiExMRHR0dFX/XxGoxFardahsAaDweF1XQ3H4nquZxwnaoz49ucLmD64EyrLS1HZQdlulKd8TQCO5UbZLXWNRgO9Xm9bFgQBSmXraiEhIRg0aBC6dGndnWvYsGHQarXXLHW1Wo34eMf2FtBqtQ6v62o4FtdzPeNY8d4hhPj74JkHbkagr+teAMNTviYAx/K/614Pu5tfkpKSsH//fgBAXl4eYmN/eWNo4MCBKC4uRm1tLSwWC/Lz89G3b1+HAhO5usOnavF9cTUeHxXj0oVO3s3uTD01NRXZ2dlIT0+HKIpYtmwZMjMzERUVhZSUFCxYsACzZ88GAIwbN+6y0ifyFKIoYtW3RQjTqPHobb2ljkN0VXZLXS6XIyMj47L7YmJibLcnTpyIiRMntn8yIheSXVKDgydrseTeAfBT8aRd5Lp48BGRHaIoYtWuIkQG++Lh4VFSxyG6JpY6kR17tFXIO1OPuSn9oFZylk6ujaVOdA2CIOK13cXo1dkfk4f2kDoOkV0sdaJr2FFQCW1lI+bdHQsfBX9cyPXxu5ToKixWAa/vLka/rhrcOzhS6jhE14WlTnQVX+SdQ1m1HgvGxPICGOQ2WOpEV2CyCFi9pxg3RQZh7E3hUschum4sdaIr2JpzBmdqW/D0mP48lTS5FZY60a8YzFa8vacEQ3t1wp39eZk6ci8sdaJf+ejgaZxvNGDBmFjO0sntsNSJ/ofeaMG670owom9n3B4TJnUcohtm99wvRN7k/QOncFFnwjtj+ksdhcghLHWiS3QmKzZ8fxopcV2RFNVJ6jhEDuHmF6JLPvu5AY0GC+al8vTR5L5Y6kQAavUmbDvegAmDwjGwe7DUcYgcxlInArD++1IYrSLmc5ZObo6lTl6vqtGADw6cwuhoDfp2DZQ6DlGbsNTJ663ZVwKrIGLqYL45Su6Pe7+QV6uoa8a/Dp3GQ8N6IjKIPw7k/jhTJ6/29p4SyGQyzE3pK3UUonbBUievdfKiHp/8VIFpw6MQEewndRyidsFSJ6+1OqsYKoUcT9zJWTp5DpY6eaWi803Ynn8OM0f0RpdAtdRxiNoNS5280uu7i6BRKfGnkX2kjkLUrljq5HWOVTTg258vYHZyH4T4q6SOQ9SuWOrkdVbtKkKIvw8eu6O31FGI2h1LnbzKwbIafF9cjSfujEGgr4/UcYjaHUudvIYoinj12yJ0C1Ljkdt6Sx2HyClY6uQ19hVV4Uh5Heam9IOvj0LqOEROwVInryAIIl79thi9OvtjyrCeUschchqWOnmFr45VQlvZiPmpsfBR8NuePBe/u8njma0CXt9VhLjwQNybECl1HCKnslvqgiBg8eLFSEtLw4wZM1BeXn7F58yePRsff/yxU0IStcUnORU4VdOMp8f0h1wukzoOkVPZLfWsrCyYTCZs3rwZCxYswIoVK37znNWrV6OhocEpAYnawmC24s2sE0iKCkFKfFep4xA5nd1Sz8nJQXJyMgAgMTERBQUFlz2+c+dOyGQyjBw50jkJidrgn/8px/lGAxaOjYNMxlk6eT67VwXQ6XTQaDS2ZYVCAYvFAqVSieLiYnz11Vd46623sGbNmut6QaPRCK1W61BYg8Hg8LquhmNxPr1JwFtZp5EU6YcQUxW02qprPt9Vx+EIjsU1dcRY7Ja6RqOBXq+3LQuCAKWydbVt27bhwoULePTRR3H27Fn4+Pige/fu15y1q9VqxMfHOxRWq9U6vK6r4Vicb3VWMRqNApZMSkJ8jxC7z3fVcTiCY3FNbRnL9f4ysFvqSUlJ2LdvHyZMmIC8vDzExv5ytfVFixbZbr/99tsICwvjZhhyCdVNRmzcX4bxA8ORcB2FTuQp7JZ6amoqsrOzkZ6eDlEUsWzZMmRmZiIqKgopKSkdkZHohr25pxhGi4BF4+KkjkLUoeyWulwuR0ZGxmX3xcTE/OZ5f/nLX9ovFVEblFTp8PGhM5g+PArRYQFSxyHqUDz4iDzOyp2F8PNRYG5KP6mjEHU4ljp5lEMna7H7+AXMuTMGnTW8TB15H5Y6eQxRFPHyDi3Cg3zx2IhoqeMQSYKlTh7j62OVyD9Tj/ljYuGn4ql1yTux1MkjGC1WvLKz9aRdk5N6SB2HSDIsdfII/zhQjtO1zXh2fBwUPGkXeTGWOrm96iYj3tpzAnf274I7+/OkXeTdWOrk9lZ9W4QWsxUv3jNA6ihEkmOpk1s7VtGALTlnMPP23ojporG/ApGHY6mT2xJFEUu+/BmdA1SYezcPNCICWOrkxrbnn0NOeR0Wju2PIF8fqeMQuQSWOrmlZpMFy3cUYlD3YDw0tKfUcYhcht0TehG5onXfleJ8owFrpg3hdUeJ/gdn6uR2Tl7UY8P+MtyfGImhvUKljkPkUljq5FZEUcQL245BrZDjrxM842o4RO2JpU5uZVveWWSX1GDR+Dh0DfKVOg6Ry2Gpk9uobzZh6VdaJPYMwbRboqSOQ+SS+EYpuY2VOwtR32LGhw8M4pujRFfBmTq5hcOnavHxoTOYdUc0BkQGSR2HyGWx1MnlmSwC/vr5MXQP8cNTPHKU6Jq4+YVc3sYfylB8QYd3HxkGfxW/ZYmuhTN1cmlF55vwZtYJTBwUgbsHdJM6DpHLY6mTy7JYBSz8JB+Bvkpk3H+T1HGI3AL/liWXtWF/GY5WNGDttCR01qiljkPkFjhTJ5dUfOGXzS4TBkVIHYfIbbDUyeUYLVY8tSmPm12IHMDNL+RyXt9VjOOVjXj3kWHc7EJ0gzhTJ5dyoOQi3vmhDNOGR3FvFyIHsNTJZdQ3mzB/Sz6iwwLwwkReRJrIEdz8Qi5BFEU88+lRXNQZ8fkjI+CnUkgdicgtcaZOLuG97FP49ucLeGZcHAb1CJY6DpHbYqmT5H46XYflO7RIHdANs5OjpY5D5Nbsbn4RBAFLlixBUVERVCoVli5dil69etkef//99/H1118DAEaNGoU///nPzktLHqdOb8Jf/pWL8GBfrHpwMGQynlKXqC3sztSzsrJgMpmwefNmLFiwACtWrLA9dubMGWzfvh2bNm3C5s2b8eOPP6KwsNCpgclzWAURT23OQ3WTEWunJSHY30fqSERuz+5MPScnB8nJyQCAxMREFBQU2B4LDw/Hu+++C4Wi9U0ti8UCtZr7FdP1eeXbQnxfXI2XHxiIhB4hUsch8gh2S12n00Gj0diWFQoFLBYLlEolfHx8EBoaClEU8corr2DAgAGIjr72NlGj0QitVutQWIPB4PC6rsbbx7KvTIcNP1RhYv8gJAU1u8T/hbd/TVwVx3Jj7Ja6RqOBXq+3LQuCAKXyl9WMRiOef/55BAQE4KWXXrL7gmq1GvHxjl0FXqvVOryuq/HmsRyraMCb/z6AW3qH4o0Zw6FSusb79d78NXFlHMsv614Puz9NSUlJ2L9/PwAgLy8PsbGxtsdEUcQTTzyB/v37IyMjw7YZhuhqztW3YPY/DqNzgAprpye5TKETeQq7M/XU1FRkZ2cjPT0doihi2bJlyMzMRFRUFARBwKFDh2AymfDDDz8AAObPn48hQ4Y4PTi5n0aDGb/PPIxmoxVb59yGMJ7Xhajd2S11uVyOjIyMy+6LiYmx3T527Fj7pyKPY7IImPPPHJRW6/DBY7cgLpwXjyZyBp4mgJxOEFpPAZBdUoPXHhqMEX3DpI5E5LG4QZOcShRFLN5egM9zz+LpMbGYPLSH1JGIPBpLnZxGFEWs+KYQ//zPafxpZB88Obqv1JGIPB5LnZzmrT0l2LC/DNNvjcKz4+N4CgCiDsBt6tTuRFHEG7uL8dbeEkxK6o6M+way0Ik6CEud2pUoilixsxAbvi/DlGE9sHxSAuRyFjpRR2GpU7sRBBEZXx3H+wdOYfqtUci4byALnaiDsdSpXRjMVizYmo+vj1Zi9h3R+OvEeG5yIZIAS53arMloxSPvHcKhk7V4fkIc/pDch4VOJBGWOrVJeY0eT39zDpU6C95MT8T9id2ljkTk1Vjq5LAfTlTjz//KhSBY8cFjt+D2GB4pSiQ1ljrdMFEU8fcfT2LZDi36dQ3EMyNCWOhELoKlTjekvtmEhZ8cxe7jFzB+YDhWPTQYp8tOSB2LiC5hqdN1O3KqFnM/zkW1zogX7xmAx0b05huiRC6GpU52GS1WrM46gQ3fl6JnqD8+nXM7rylK5KJY6nRN+Wfq8fTWfJyo0iFtWE+8cE88An19pI5FRFfBUqcrajKY8fruYnxw4BS6Bvoi8/c3Y3T/rlLHIiI7WOp0GVEUsT3/HJZ+rcVFnRFTb4nConFxCPbj7JzIHbDUyebwqVos26FF7ul6JPQIxruPDMPgntx2TuROWOqE4+casTqrGLuOX0C3IDVWTh6EB4f2hIIn4yJyOyx1L5Z7ug5r9pUgS1sFjVqJp8fE4rE7ouGv4rcFkbviT6+XEUURB0prsPa7EmSX1CDE3wfzU2Px6G29EezP7eZE7o6l7iUaWsz4NKcCHx0sR2m1HmEaNZ6fEIdpw3shQM1vAyJPwZ9mDyYIIn46XYetRyrwRf5ZGMwCEnuGYNVDg3FPQgR8fRRSRySidsZS9zCiKOJ4ZSO255/DV/mVOFvfAj8fBR4Y0h3ThvfCwO7BUkckIidiqXsAi1VATnkd9hZWIUt7AaXVeijlMiT3C8PTY2OROiAcGm5iIfIK/El3U2dqm/Gfshr8cOIiviuqQqPBAh+FDLdEh+L3I6IxYVAEQgNUUsckog7GUncDVkHEyYs65J6ux8GTtfhPWQ0q6loAAJ0DVBhzUzhS4rrijn5hPC8LkZdjqbsYQRBRUdeCo2frcbSiAfln6lFwtgF6kxUA0MnfB8OjO2P2HdG4NaYzYrsGQs6DhIjoEpa6RCyCiJMX9Sip0uFEVRNKLuhwokqH0modmi8VuEohR3xkECYP7YFB3YMxuGcI+nbRsMSJ6KpY6k5iMFtR3WRERV0LKuqaUVHXgjOXPp6ta0FlQwsE8aTt+RHBvujbVYO0m3sitlsgBnUPRmy3QKiUcglHQUTuxm6pC4KAJUuWoKioCCqVCkuXLkWvXr1sj2/ZsgWbNm2CUqnEnDlzMHr0aKcGloIoimgxW1HfbEZDS+u/+mYzGlvMqNGbUN1kRLXOiOomA6qajKhuMqLJYLnsc8hkQHiQL3p28sfw6FCorXokxUahb1cN+nbVcFs4EbULu6WelZUFk8mEzZs3Iy8vDytWrMC6desAANXV1fjwww/x6aefwmg0YurUqRgxYgRUqo7d60IURZisAkyW1n9mq9h622qF0fLL/c1mK1pMVuiNFrSYrdAbrWgxWaA3WdFssqLZZLnsY2PLLyVutopXfX2NWokugWp00agRHx6Ekf3UtuXIED/0DPVDRLDfZbNurVaL+PieHfHfQ0RexG6p5+TkIDk5GQCQmJiIgoIC22NHjx7FkCFDoFKpoFKpEBUVhcLCQiQkJLR70KMV9Zi7vQLWLyttJW0rcKvQps8doFLAT6VEgFoBPx8FAtRKaNRKRAT7IthPhWA/H4T4+7R+9Gv9GHxpOTRAxRNgEZHLsNtGOp0OGo3GtqxQKGCxWKBUKqHT6RAYGGh7LCAgADqd7pqfz2g0QqvV3nDQqkYzegUpoFAo4KNQwkcug49CBqUc8FHIbMs+chmUv1pu/Qj4KuXw9ZHDVymDn1IOXx8Z1ArZdVw8WQRguvQPgBlAA9DUADTd8EhaGQwGh/4fXJGnjMVTxgFwLK6qI8Zit9Q1Gg30er1tWRAEKJXKKz6m1+svK/krUavViI+Pv+Gg8QC6B2kdWtcVtW5+4VhciaeMA+BYXFVbxnK9vwzs7lqRlJSE/fv3AwDy8vIQGxtreywhIQE5OTkwGo1oampCaWnpZY8TEVHHsjtTT01NRXZ2NtLT0yGKIpYtW4bMzExERUUhJSUFM2bMwNSpUyGKIubNmwe1Wt0RuYmI6ArslrpcLkdGRsZl98XExNhuT5kyBVOmTGn/ZEREdMN4ZAsRkQdhqRMReRCWOhGRB2GpExF5EJY6EZEHkYmiePWTmjhBXl4ed3skIrpBRqMRiYmJdp/X4aVORETOw80vREQehKVORORBWOpERB6EpU5E5EFY6kREHsStSt1qtWLp0qVIT0/HpEmTsG/fPqkjtVlpaSmGDh0Ko9EodRSHNTU14fHHH8f06dORlpaG3NxcqSPdEEEQsHjxYqSlpWHGjBkoLy+XOpLDzGYzFi5ciKlTp+LBBx/Enj17pI7UJjU1NRg1ahRKS0uljtImGzZsQFpaGiZNmoStW7c69bXc6jpsX3zxBSwWCzZt2oQLFy7gm2++kTpSm+h0OqxcubLDr+na3jIzM3Hrrbdi5syZKCsrw4IFC/D5559LHeu6Xes6vO5m+/btCAkJwaslGbcPAAAC1ElEQVSvvoq6ujo88MADSElJkTqWQ8xmMxYvXgxfX1+po7TJwYMHkZubi48//hgtLS147733nPp6blXqP/74I2JjY/HHP/4RoijixRdflDqSw/6bf/78+XjiiSekjtMmM2fOtP1islqtbndw2bWuw+tuxo0bh7Fjx9qWFQqFhGnaZuXKlUhPT8c777wjdZQ2+W9vPfnkk9DpdFi0aJFTX89lS33r1q344IMPLruvU6dOUKvV2LBhAw4fPoznnnsOH330kUQJr9+VxhIZGYkJEyYgLi5OolSOudJYli1bhoSEBFRXV2PhwoV4/vnnJUrnmGtdh9fdBAQEAGgd09y5c/HUU09JnMgxn332GUJDQ5GcnOz2pV5XV4dz585h/fr1qKiowJw5c7Bz587ruDayY9zqiNJ58+ZdNhMZMWIEsrOzJU7lmNTUVISHhwNoPXVCQkKCW/yCupqioiLMnz8fixYtwqhRo6SOc0OWL1+OwYMHY8KECQCAkSNH2i7h6I4qKyvx5JNP2raru6Np06ZBJmu9KLxWq0Xv3r2xbt06dOnSRepoN2zVqlUIDQ3FY489BgC47777kJmZic6dOzvnBUU38uGHH4rPPfecKIqiqNVqxcmTJ0ucqH2MHj1aNBgMUsdw2IkTJ8SxY8eKWq1W6igO2blzp/jMM8+IoiiKubm54qxZsyRO5Ljq6mpx3Lhx4oEDB6SO0m6mT58ulpSUSB3DYXv37hVnzpwpCoIgnj9/Xrz77rtFi8XitNdzq78vp0yZgpdeeglTpkyBKIr429/+JnUkAvDaa6/BZDLh5ZdfBgBoNBq3eqPxStfhdVfr169HY2Mj1q5di7Vr1wIANm7c6PZvNrqz0aNH4/Dhw3jwwQchiiIWL17s1Pc63GrzCxERXZtb7adORETXxlInIvIgLHUiIg/CUici8iAsdSIiD8JSJyLyICx1IiIPwlInIvIg/w9790UgCev+7gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x=np.linspace(-6,6,200)\n",
    "y=1/(1+np.exp(-x))\n",
    "\n",
    "plt.plot(x,y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到,当z=0时,g(z)=0.5,当z>0时,g(z)>0.5,当z越来越大时,g(z)无限接近于1.当z越来越小的时候,起无限接近于0.这正是我们想要的针对二元分类算法的预测函数.  \n",
    "结合线性回归函数的预测函数,$ h_0(x)=\\theta^Tx $,假设另$ z(x)=\\theta^Tx $,则逻辑回归算法的预测函数如下:\n",
    "$$\n",
    "h_0(x)=g(z)=g(\\theta^Tx)=\\frac{1}{1+e^{-\\theta^Tx}}\n",
    "$$\n",
    "\n",
    "$ h_0(x)$表示在输入值为x,参数为$ \\theta $的条件下Y=1的概率.用概率论的公式可以写成:\n",
    "$$\n",
    " h_0(x)=P(y=1|x,\\theta)\n",
    "$$\n",
    "> 在输入x及参数$ \\theta$的条件下,y=1的概率.这是一个条件概率公式.对于二元分类法来说,这是个非黑即白的世界."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 判定边界  \n",
    "逻辑回归算法的预测函数由下面两个公式给出:\n",
    "$$\n",
    "h_0(x)=g(\\theta^Tx)\n",
    "$$\n",
    "$$\n",
    "g(x)=\\frac{1}{1+e^{-z}}\n",
    "$$\n",
    "假定y=1的判定条件是$h_0(x)\\geqslant0.5$,y=0的判定条件是$h_0(x)\\leqslant0.5$,则可以推导出y=1的判定条件就是$\\theta^Tx\\geqslant 0$,y=0的判定条件就是$\\theta^Tx<0$.所以$\\theta^Tx=0$就是我们的判定边界."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. 成本函数  \n",
    "线性模型的成本函数去推导逻辑回归的成本函数是不合适的,因为这样出来的函数太复杂.为了容易找到成本函数的最小值.我们分成y=1和y=0两种情况来分辨考虑其预测值和真实值的误差.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "成本函数:\n",
    "$$\n",
    "J(\\theta)=-\\frac{1}{m}\\left[\\sum_{i=1}^my^{(i)}\\log(h_0(x^{i}))+(1-y^{i})log(1-h_0(x^(i))\\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4.梯度下降算法**  \n",
    "和线性回归类似,我们也可以使用梯度下降算法来求解逻辑回归模型参数.根据梯度下降算法的定义.可以得到:\n",
    "$$\n",
    "\\theta_j=\\theta_j-\\alpha\\frac{\\delta}{\\delta\\theta_j}J(\\theta)\n",
    "$$\n",
    "这里的关键是求解成本函数的偏导数,最终推倒出来的梯度下降算法的公式为:\n",
    "$$\n",
    "\\theta_j=\\theta_j-\\alpha\\frac{1}{m}\\sum_{i=1}{m}(h_0(x^{i}-y^{(i)})x_j^{(i)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.多元分类\n",
    "\n",
    "逻辑回归模型除了可以解决二元分类问题,还可以解决多元分类的问题.针对多元分类问题:y={0,1,2...n},总共有n+1个类别.其解决的基本思路是,首先把问题转换为二元分类问题.即y=0是一个类别,y={1,2,3...n}是一个类别.然后分别计算两个类别的概率.接着把y=1作为一个类别,剩余的做另外一个类别,以此推广.总共需要n+1个预测函数.预测出来的概率最高的那个类别,就是样本所属的类别."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.正则化\n",
    "正则化也是用来解决模型过拟合的一个办法:\n",
    "* 保留所有的特征,减小特征的权重$ \\theta_j $的值,确保所有特征对预测值都有少量的贡献.\n",
    "* 当每个特征$x_i$对预测值y都有少量贡献时,这样的模型可以良好的工作,这就是正则化的目的,可以用它来解决特征过多时,过拟合的问题."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 线性回归模型正则化\n",
    "线性回归模型的成本函数正则化:\n",
    "$$\n",
    "J(\\theta)=\\frac{1}{2m} \\left[\\sum_{i=1}^m(h_0(x^{i}-y^{i})^2 \\right]+\\lambda\\sum_{i=1}^n\\theta_j^2\n",
    "$$\n",
    "- 公式的前半部分是线性回归模型的成本函数,也称为预测值和实际值的误差.\n",
    "- 后半部分为加入的正则项.\n",
    "    - $\\lambda$的值有两个目的:纪要维持对训练样本的拟合,又要避免对训练样本的过拟合.如果$\\lambda$的值太大,则能确保不出现过拟合,但可能也会导致出现对训练样本的欠拟合.\n",
    "    \n",
    "从数学的角度看,成本函数增加了一个正则项后,成本函数不再唯一的由预测值和真实值的误差决定,还和参数$\\theta$的大小有关.有了这个限制后,要实现成本函数的最小目的,其就不能随意取值了.比如$\\theta$值较大的时候,会导致预测值和真实值的误差值很小,但会导致$\\theta_j^2$很大.最终结果还是会导致成本函数太大.这样通过条件参数$\\lambda$,就可以控制正则项的权重.从而避免线性回归函数的过拟合.\n",
    "\n",
    "### 3.2 逻辑回归模型正则化\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.算法参数\n",
    "\n",
    "1. 正则项权重\n",
    "2. L1/L2范数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
